Feature/large pipeline #2706

ameenkhan07 · 2021-10-14T23:55:25Z

Issue #, if available:
Add support for large pipeline definition and step parallelism configuration

Description of changes:
Updated the create/update pipeline to upload large pipeline definition (greater than 100Kb) to S3 bucket and replaces PipelineDefinition with PipelineDefinitionS3Location

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backword compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…intion

…hods

sagemaker-bot · 2021-10-14T23:59:02Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: c30bb0f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-15T00:35:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: c30bb0f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-15T00:50:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: c30bb0f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navaj0 · 2021-10-15T23:31:08Z

src/sagemaker/workflow/pipeline.py

@@ -102,37 +105,65 @@ def create(
            description (str): A description of the pipeline.
            tags (List[Dict[str, str]]): A list of {"Key": "string", "Value": "string"} dicts as
                tags.
+            parallelism_config (Optional[Config for parallel steps, Parallelism configuration that


(Optional[ParallelismConfiguration]): parallelism configuration that applied to each of the executions of the pipeline.

Can you update the doc string in other places as well.

navaj0 · 2021-10-15T23:34:43Z

src/sagemaker/workflow/pipeline.py

@@ -223,6 +265,8 @@ def start(
                pipeline parameters.
            execution_display_name (str): The display name of the pipeline execution.
            execution_description (str): A description of the execution.
+            parallelism_config (Optional[ParallelismConfiguration]): Config for parallel steps, that


(Optional[ParallelismConfiguration]): parallelism configuration that applied to the newly started execution. It takes precedence over the parallelism configuration of the parent pipeline.

navaj0 · 2021-10-15T23:40:43Z

tests/integ/test_workflow.py

+        except Exception:
+            pass
+
+def test_create_parallelism_config(sagemaker_session, role, pipeline_name, region_name):


test_create_and_update_with_parallelism_config

navaj0 · 2021-10-15T23:44:36Z

tests/integ/test_workflow.py

+    try:
+        response = pipeline.create(role, parallelism_config={"MaxParallelExecutionSteps": 50})
+        create_arn = response["PipelineArn"]
+        assert re.match(


Also do a describe call to verify the configuration

navaj0 · 2021-10-15T23:44:46Z

tests/integ/test_workflow.py

+        pipeline.parameters = [ParameterInteger(name="InstanceCount", default_value=1)]
+        response = pipeline.update(role, parallelism_config={"MaxParallelExecutionSteps": 50})
+        update_arn = response["PipelineArn"]
+        assert re.match(


Also do a describe call to verify the configuration

navaj0 · 2021-10-15T23:45:57Z

tests/integ/test_workflow.py

+        pipeline.parameters = [ParameterInteger(name="InstanceCount", default_value=1)]
+        response = pipeline.update(role)
+        update_arn = response["PipelineArn"]
+        assert re.match(


Do a describe call to verify the configuration

sagemaker-bot · 2021-10-19T05:35:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: e777674
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-19T06:22:46Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: e777674
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-19T06:31:05Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: e777674
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-27T20:30:25Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: bc7573d
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-10-27T21:26:43Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: bc7573d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala

Please fix the unit test black-check errors.

ameenkhan07 · 2021-11-05T00:23:00Z

Please fix the unit test black-check errors.

Fixed formatting error

codecov-commenter · 2021-11-05T00:34:06Z

Codecov Report

❗ No coverage uploaded for pull request base (master@4571ea2). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2706   +/-   ##
=========================================
  Coverage          ?   88.80%           
=========================================
  Files             ?      168           
  Lines             ?    14724           
  Branches          ?        0           
=========================================
  Hits              ?    13076           
  Misses            ?     1648           
  Partials          ?        0

Impacted Files	Coverage Δ
src/sagemaker/_studio.py	`91.30% <ø> (ø)`
src/sagemaker/algorithm.py	`90.56% <ø> (ø)`
src/sagemaker/amazon/amazon_estimator.py	`89.36% <ø> (ø)`
src/sagemaker/amazon/common.py	`93.87% <ø> (ø)`
src/sagemaker/amazon/factorization_machines.py	`100.00% <ø> (ø)`
src/sagemaker/amazon/hyperparameter.py	`97.36% <ø> (ø)`
src/sagemaker/amazon/ipinsights.py	`100.00% <ø> (ø)`
src/sagemaker/amazon/kmeans.py	`100.00% <ø> (ø)`
src/sagemaker/amazon/knn.py	`100.00% <ø> (ø)`
src/sagemaker/amazon/lda.py	`97.61% <ø> (ø)`
... and 61 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4571ea2...7370871. Read the comment docs.

sagemaker-bot · 2021-11-05T00:36:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 7370871
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-11-05T01:17:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 7370871
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Co-authored-by: Basil Beirouti <[email protected]>

The base branch was changed.

sagemaker-bot · 2021-12-06T23:29:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a8d42ca
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-06T23:29:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 458d4f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-07T00:09:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 458d4f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-07T00:09:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a8d42ca
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2021-12-07T00:31:57Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 458d4f8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-05T01:12:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 458d4f8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-01-05T02:10:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 458d4f8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

shreyapandit · 2022-01-13T23:25:36Z

Please fix conflicts and address feedback to make this eligible for review again

ahsan-z-khan

closing this. Please reopen when working on it.

ameenkhan07 · 2022-02-07T21:10:25Z

Changes merged in: #2825

Ameen Khan added 2 commits October 14, 2021 23:46

feature: Updated create/update pipeline to support large pipeline def…

103b7e9

…intion

feature: Added parallelism config to create/update/start pipeline met…

c30bb0f

…hods

navaj0 reviewed Oct 15, 2021

View reviewed changes

Updated docstring, added describe call to check config

e777674

shreyapandit requested review from shreyapandit and mufaddal-rohawala October 19, 2021 21:12

staubhp previously approved these changes Oct 27, 2021

View reviewed changes

Merge branch 'master' into feature/large_pipeline

bc7573d

mufaddal-rohawala requested changes Oct 28, 2021

View reviewed changes

Fixed formatting issues

7370871

ameenkhan07 dismissed staubhp’s stale review via 7370871 November 5, 2021 00:22

feature: allow conditional parellel builds (aws#2727)

53061e8

navaj0 previously approved these changes Dec 6, 2021

View reviewed changes

BasilBeirouti and others added 2 commits December 6, 2021 14:56

fix endpoint bug (aws#2772)

1ff8ae3

Co-authored-by: Basil Beirouti <[email protected]>

Merge branch 'master' into feature/large_pipeline

a8d42ca

ahsan-z-khan changed the base branch from master to dev December 6, 2021 23:12

Merge branch 'dev' into feature/large_pipeline

458d4f8

shreyapandit force-pushed the dev branch from 2f7c40a to 0bae071 Compare December 15, 2021 19:38

mufaddal-rohawala force-pushed the dev branch from beeabbd to b026769 Compare January 4, 2022 17:09

shreyapandit force-pushed the dev branch from 1c039c8 to 7206b9e Compare January 10, 2022 23:05

shreyapandit added the needs contributor changes label Jan 13, 2022

shreyapandit marked this pull request as draft January 13, 2022 23:25

shreyapandit force-pushed the dev branch from 903205c to b796a00 Compare January 18, 2022 20:09

ahsan-z-khan reviewed Jan 22, 2022

View reviewed changes

ahsan-z-khan closed this Jan 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/large pipeline #2706

Feature/large pipeline #2706

ameenkhan07 commented Oct 14, 2021

sagemaker-bot commented Oct 14, 2021

sagemaker-bot commented Oct 15, 2021

sagemaker-bot commented Oct 15, 2021

navaj0 Oct 15, 2021 •

edited

Loading

navaj0 Oct 15, 2021

navaj0 Oct 15, 2021

navaj0 Oct 15, 2021

navaj0 Oct 15, 2021

navaj0 Oct 15, 2021

sagemaker-bot commented Oct 19, 2021

sagemaker-bot commented Oct 19, 2021

sagemaker-bot commented Oct 19, 2021

sagemaker-bot commented Oct 27, 2021

sagemaker-bot commented Oct 27, 2021

mufaddal-rohawala left a comment

ameenkhan07 commented Nov 5, 2021

codecov-commenter commented Nov 5, 2021

sagemaker-bot commented Nov 5, 2021

sagemaker-bot commented Nov 5, 2021

sagemaker-bot commented Dec 6, 2021

sagemaker-bot commented Dec 6, 2021

sagemaker-bot commented Dec 7, 2021

sagemaker-bot commented Dec 7, 2021

sagemaker-bot commented Dec 7, 2021

sagemaker-bot commented Jan 5, 2022

sagemaker-bot commented Jan 5, 2022

shreyapandit commented Jan 13, 2022

ahsan-z-khan left a comment

ameenkhan07 commented Feb 7, 2022

Feature/large pipeline #2706

Feature/large pipeline #2706

Conversation

ameenkhan07 commented Oct 14, 2021

Merge Checklist

General

Tests

sagemaker-bot commented Oct 14, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 15, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 15, 2021

AWS CodeBuild CI Report

navaj0 Oct 15, 2021 • edited Loading

Choose a reason for hiding this comment

navaj0 Oct 15, 2021

Choose a reason for hiding this comment

navaj0 Oct 15, 2021

Choose a reason for hiding this comment

navaj0 Oct 15, 2021

Choose a reason for hiding this comment

navaj0 Oct 15, 2021

Choose a reason for hiding this comment

navaj0 Oct 15, 2021

Choose a reason for hiding this comment

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 19, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 27, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Oct 27, 2021

AWS CodeBuild CI Report

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

ameenkhan07 commented Nov 5, 2021

codecov-commenter commented Nov 5, 2021

Codecov Report

sagemaker-bot commented Nov 5, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Nov 5, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 6, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 6, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Dec 7, 2021

AWS CodeBuild CI Report

sagemaker-bot commented Jan 5, 2022

AWS CodeBuild CI Report

sagemaker-bot commented Jan 5, 2022

AWS CodeBuild CI Report

shreyapandit commented Jan 13, 2022

ahsan-z-khan left a comment

Choose a reason for hiding this comment

ameenkhan07 commented Feb 7, 2022

navaj0 Oct 15, 2021 •

edited

Loading